In 1984, the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation issued a requirement for frontal crash protection of automobile occupants. This regulation mandated "passive occupant restraints" for all passenger cars by 1992. A more recent regulation requires both driver and passenger side airbags on all passenger cars and light trucks by 1998. In addition, the demand for airbags is accelerating in both Europe and Japan and it is expected that, now virtually all vehicles produced in these areas (36 million vehicles) are equipped and eventually worldwide (50 million vehicles) will be equipped with airbags as standard equipment.
Whereas thousands of lives have been saved by airbags, significant improvements can be made. As discussed in detail in U.S. Pat. No. 5,653,462 referenced above, and included herein by reference, for a variety of reasons, vehicle occupants can be or get too close to the airbag before it deploys and can be seriously injured or killed by the deployment of the airbag.
Also, a child in a rear facing child seat, which is placed on the right front passenger seat, is in danger of being seriously injured if the passenger airbag deploys. This has now become an industry-wide concern and the US automobile industry is urgently searching for an easy, economical solution, which will prevent the deployment of the passenger side airbag if a rear facing child seat is present. An improvement on the invention disclosed in the above-referenced patent application, as will be disclosed in greater detail below, includes more sophisticated means to identify objects within the passenger compartment and will solve this problem.
Initially, these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded deployments when a seat is unoccupied. Airbags are now under development to protect rear seat occupants in frontal and side vehicle crashes. A system will therefore be needed to detect the presence and position of occupants, to determine if they are out-of-position, and type to identify the presence of a rear facing child seat in the rear seat, for example. Future automobiles can be expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident. The improvements described below minimize this cost by not deploying an airbag for a seat that is not occupied by a human being. An occupying item of a seat may be a living occupant such as a human being or dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
A device to monitor the vehicle interior and identify its contents is needed to solve these and many other problems. For example, once a Vehicle Interior Identification and Monitoring System (VIMS) for identifying and monitoring the contents of a vehicle is in place, many other products become possible including the following:
1. Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system of this invention will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. PA1 2. Side impact airbag systems began appearing on 1995 vehicles. The danger of deployment induced injuries exist for side impact airbags as they now do for frontal impact airbags. A child with his head against the airbag is such an example. The system of this invention will minimize such injuries. PA1 3. Future vehicles may be provided with a standard cellular phone as well as the Global Positioning System (GPS), an automobile navigation or location system, which is now available on several vehicle models. In the event of an accident, the phone may automatically call 911 for emergency assistance and report the exact position of the vehicle. If the vehicle also has a system as described below for monitoring each seat location, the number and perhaps the condition of the occupants could also be reported. In that way, the emergency service (EMS) would know what equipment and how many ambulances to send to the accident site. PA1 4. Vehicle entertainment system engineers have stated that the quality of the sound in the vehicle could be improved if the number, size and location of occupants and other objects were known. This information can be provided by the vehicle interior identification and monitoring system of this invention. PA1 5. Similarly to the entertainment system, the heating, ventilation and air conditioning system (HVAC) could be improved if the number, attributes and location of vehicle occupants were known. This can be used to provide a climate control system tailored to each occupant, for example, or the system can be turned off for certain seat locations if there are no occupants present at those locations. PA1 6. In some cases, the position of a particular part of the occupant is of interest such as: (a) the occupant's hand or arm and whether it is in the path of a closing window so that the motion of the window needs to be stopped; (b) the position of the occupant's shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant; or, (c) the position of the rear of the occupant's head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts. PA1 1. To enable the presence, position and type of occupying item in a passenger compartment to be detected even with the presence of thermal gradients. PA1 2. To recognize the presence of a human on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system, heating and air conditioning system, or entertainment system, among others. PA1 3. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her position and to use this position information to affect the operation of another vehicle system. PA1 4. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her velocity relative to the passenger compartment and to use this velocity information to affect the operation of another vehicle system. PA1 5. To determine the position of a seat in the vehicle using sensors remote from the seat and to use that information in conjunction with a memory system and appropriate actuators to position the seat to a predetermined location. PA1 6. To determine the position, velocity or size of an occupant in a motor vehicle and to utilize this information to control the rate of gas generation, or the amount of gas generated, by an airbag inflator system or to control the amount of gas flowing into or out of the airbag. PA1 7. To determine the fact that an occupant is not restrained by a seatbelt and therefore to modify the characteristics of the airbag system. PA1 8. To determine the presence or position of rear seated occupants in the vehicle and to use this information to affect the operation of a rear seat protection airbag for frontal impacts. PA1 9. To determine the presence or position of occupants relative to the side impact airbag systems and to use this information to affect the operation of a side impact protection airbag system. PA1 10. To determine the presence of an occupant's hand or other object in the path of a closing window and to affect the window closing system. PA1 11. To determine the position of the shoulder of a vehicle occupant and to use that information to control the seatbelt anchorage point. PA1 12. To determine the position of the rear of an occupant's head and to use that information to control the position of the headrest. PA1 13. To recognize the presence of a rear facing child seat on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system. PA1 14. To determine the total number of occupants of a vehicle and in the event of an accident to transmit that information, as well as other information such as the condition of the occupants, to a receiver remote from the vehicle. PA1 15. To affect the vehicle entertainment system based on a determination of the size or location of various occupants or other objects within the vehicle passenger compartment. PA1 16. To affect the vehicle heating, ventilation and air conditioning system based on a determination of the number, size and location of various occupants or other objects within the vehicle passenger compartment. PA1 17. To provide a vehicle interior monitoring system for determining the location of occupants within the vehicle and to include within the same system various electronics for controlling an airbag system. PA1 18. To determine the approximate location of the eyes of a driver and to use that information to control the position of the rear view mirrors of the vehicle. PA1 19. To monitor the position of the head of the vehicle driver and determine whether the driver is falling asleep or otherwise impaired and likely to lose control of the vehicle and to use that information to affect another vehicle system. PA1 20. To determine the location of the eyes of a vehicle occupant and the direction of a light source such as the headlights of an oncoming vehicle or the sun and to cause a filter to be placed in such a manner as to reduce the intensity of the light striking the eyes of the occupant. PA1 21. To determine the location of the eyes of a vehicle occupant and the direction of a light source such as the headlights of a rear approaching vehicle or the sun and to cause a filter to be placed in such a manner as to reduce the intensity of the light reflected from the rear view mirrors and striking the eyes of the occupant. PA1 22. To determine the location of the ears of a vehicle occupant and to use that information to control the entertainment system so as to improve the quality of the sound reaching the occupant's ears through such methods as noise canceling sound. PA1 23. To recognize a particular driver based on such factors as physical appearance or other attributes and to use this information to control another vehicle system such as a security system, seat adjustment, or maximum permitted vehicle velocity, among others. PA1 1. To provide a method for reducing the effects of thermal gradients that occur when the sun beats down on a closed vehicle or from the operation of the heater or air conditioner, such gradients causing the ultrasonic waves to be diffracted and thereby changing the received wave pattern. PA1 2. To provide a reliable method using a single transducer for both sending and receiving ultrasonic waves while permitting objects to be detected that are less than 4 inches from the transducer. PA1 3. To provide a reliable method for dynamically determining the location of a vehicle occupant who is moving toward the airbag module due to vehicle decelerations caused by, for example, pre-crash braking and to use this information to control another vehicle system such as the airbag system. PA1 4. To provide a reliable method for compensating for the effects of the change in the speed of sound due to temperature changes within the vehicle, such method based on the variation of a measurable property of the transducer such as its capacitance, inductance or natural frequency with temperature. PA1 5. To provide a reliable method for determining in a timely manner, such as every 10-20 milliseconds, that an occupant is out of position, or will become out of position, and likely to be injured by a deploying airbag and to then output a signal to suppress the deployment of the airbag and to do so in sufficient time that the airbag deployment can be suppressed even in the case of a poorly designed or malfunctioning crash sensor which triggers late on a short duration crash. PA1 6. To provide a method of controlling the wave pattern emitted from the transducer assembly so as to more precisely illuminate the area of interest. PA1 7. To provide apparatus which permits speed of sound compensation to be achieved even when each transducer in the system operates at a different tuned frequency. PA1 8. To provide apparatus which permits detecting objects that are very close to the transducer assembly.
The above applications illustrate the wide range of opportunities that become available if the identity and location of various objects and occupants, and some of their parts, within the vehicle were known. Once the system is operational, it would be logical for the system to also incorporate the airbag electronic sensor and diagnostics module (SDM) since it needs to interface with SDM anyway and since the two electronic circuits could share computer capabilities resulting in a significant cost saving to the auto manufacturer. For the same reasons, it would be logical for VIMS to include the side impact sensor and diagnostic system since occupant sensing systems can also be used with side airbags.
As the VIMS improves to where the exact location of the occupants ears and eyes can be determined, even more significant improvements to the entertainment system become possible through the use of noise canceling sound, and the rear view mirror can be automatically adjusted for the driver's eye location. Another example involves the monitoring of the driver's behavior over time which can be used to warn a driver if he or she is falling asleep, or to stop the vehicle if the driver loses the capacity to control it.
Using an advanced VIMS, as explained below, the position of the driver's eyes can be accurately determined and portions of the windshield can be selectively darkened to eliminate the glare from the sun or oncoming vehicle headlights. This system uses electro-chromic glass, a liquid crystal system, or other appropriate technology, and detectors to detect the direction of the offending light source. In addition to eliminating the glare, the sun visor can now also be eliminated.
1. Prior Art on Apparatus for Sensing Out-of-position Occupants and Rear Facing Child Seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, and almost 150 people have now been killed, by the deploying airbag, and thus significant improvements need to be made to eliminate these deaths and injuries. As discussed in detail in one or more of the patents and copending patent applications cross-referenced above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys, or, if unbelted, may be thrown there by the crash dynamics and can be seriously injured or killed as a result of the airbag deployment. Also, a child in a rear facing child seat which is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. "How Airbags Work" presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection is required in order to minimize the damages caused by deploying airbags.
Inflators now exist which have the capability of adjusting the amount of gas flowing into the airbag to account for the size and position of the occupant and for the severity of the accident and thereby adapt the deployment of the airbag to the occupant and accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782, among others, can control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The instant invention is an improvement on that VIMS system and uses an advanced ultrasonic system comprising two or more ultrasonic transmitters/receivers combined with a trained neural network or neural-fuzzy pattern recognition system as discussed in much greater detail below.
The automatic adjustment of the deployment rate of the airbag based on occupant identification and position and on crash severity has been termed "smart airbags". Central to the development of smart airbags is the occupant identification and position system described herein. Such smart airbag systems are described in more detail in U.S. patent application Ser. No. 08/865,525 entitled "Smart Airbag System" filed May 29, 1997 which is also included herein by reference. To complete the development, an anticipatory crash detecting system such as disclosed in U.S. patent application Ser. No. 08/247,760 filed May 23, 1994 is desirable. Prior to the implementation of anticipatory crash sensing, the use of a neural network smart crash sensor which identifies the type of crash and thus its severity based on the early part of the crash acceleration signature should be developed and thereafter implemented. U.S. Pat. No. 5,684,701 to Breed, describes a crash sensor based on neural networks. This crash sensor, as with all other crash sensors, determines whether or not the crash is of sufficient severity to require deployment of the airbag and, if so, initiates the deployment. A neural network based on a smart airbag crash sensor can also be designed to identify the crash and categorize it with regard to severity thus permitting the airbag deployment to be matched not only to the characteristics and position of the occupant but also the severity and timing of the crash itself, as more fully described in U.S. patent application Ser. No. 08/247,760.
Significant improvements were made to the art in U.S. patent application Ser. No. 08/798,029 referenced above which describes the method of placement of the transducers to increase the reliability of detecting and discriminating out-of-position occupants, empty seats, and rear facing child-seats. In order to detect occupants that are very close to the transducer in that invention, separate transducers are used for sending and receiving the ultrasonic waves. Also, although that system is capable of detecting out-of-position occupants for most real world cases, in situations where the crash sensor fails to trigger or triggers very late in a high speed crash, the system based on alternately transmitting and receiving from each location can require as much as 50 milliseconds to determine the location of an occupant which can be too slow. The use of one or two transducers for ranging during the crash, giving 10 or 20 millisecond response time, works in most cases but can be defeated if the chosen transducer is blocked by a newspaper, for example. Finally, the wide beam patterns of the transducers used in that system sometimes results in false decisions when an occupant of the rear seat is leaning forward, for example, and the system interprets that as an in position forward facing person even thought in fact it may be a rear facing child seat.
Others have also observed the need for an occupant out-of-position sensor and several methods have been disclosed in certain U.S. patents for determining the position of an occupant of a motor vehicle. Some of these systems will be discussed below and unfortunately have significant limitations.
In White et al. (U.S. Pat. No. 5,071,160), for example, a single acoustic sensor and detector is described and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors but do not disclose where they would be mounted, other than on the instrument panel below the steering wheel, or how they would be combined to uniquely monitor particular locations in the passenger compartment and to identify what is occupying those locations.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. Their use of these sensors is to measure the change in position of an occupant during a crash and use that information to assess the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in the above cross-referenced patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use a combination of ultrasonic sensors/transmitters to identify the presence of a human occupant and then to find his/her location in the passenger compartment.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. disclose only lower mounting locations of their sensors that are mounted in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant's hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver's arms are the closest objects to two of the sensors.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant's hands, as in the case where he/she is adjusting the knob on the radio, and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was moving. Such a combination, however, is not disclosed in White and would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant's head or chest. Since the sizes and driving positions of occupants are extremely varied, trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed. White et al. does not suggest the use of such neural networks.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position, or if a rear facing child seat is present. In fact, the closer that the occupant gets to the airbag the faster the inflation rate of the airbag is according to the Fujita patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat. This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant's head.
Finally, Corrado et al., in U.S. Pat. No. 5,482,314 describe a method of determining the location of an occupant based on the "fusion" of the information from an ultrasonic and a passive infrared sensor. The passive infrared sensor can be easily fooled by: (i) an occupant holding a cup of coffee near to where a rear facing child seat would be located; (ii) an increase in the ambient temperature to above body temperature; (iii) the use of a blanket to cover the occupant of a rear facing child seat; (iv) or by any other method by which the sensor is blocked such as by a newspaper or map. The ultrasonic sensor is used only in a ranging mode and therefore can only measure the distance to the closest object which may by a newspaper or balloon or the occupant's hat or hand. Since both sensor systems are easily fooled the combination is also unreliable. Nowhere in the patent does it answer the question of which sensor to believe if one says one thing and the other something else.
It is important to note that in all cases in the prior art, except those assigned to the current assignee of the instant invention, where ultrasonic sensors are used to determine displacement, only the initial return of reflected waves is used so that only the distance to the closest part of the object can be determined. In contrast, in the instant invention, the return echo pattern over several milliseconds corresponding to the entire portion of the passenger compartment volume of interest is analyzed providing distance information to many points on the items occupying the passenger compartment.
2. Definitions
The use of pattern recognition is central to the instant invention as well as those cross-referenced patent applications above, although the improvements disclosed herein may also be used in other systems and therefore this invention is not limited to systems using pattern recognition. Nowhere in the prior art, except in that assigned to the current assignee of the instant invention, is pattern recognition which is based on training, as exemplified through the use of neural networks, mentioned for use in monitoring the interior passenger compartment or exterior environments of the vehicle.
"Pattern recognition" as used herein will mean any system which processes one or more signals that are generated by an object, e.g., representative of a pattern of returned or received impulses, waves or other physical property specific to and/or characteristic of and/or representative of that object, or is modified by interacting with an object, in order to determine to which one of a set of classes that the object belongs. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed generally include a series of electrical signals coming from transducers that are sensitive to acoustic (ultrasonic) or electromagnetic radiation.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught to recognize various patterns constituted within the signals by subjecting the system to a variety of examples. The most successful such system is the neural network or neural fuzzy system. Thus, to generate the pattern recognition algorithm, test data is first obtained which constitutes a plurality of sets of returned waves, or wave patterns and other information from weight, seat position sensors etc., from an object and an indication of the identify of that object, i.e., a number of different objects are tested to obtain the unique patterns in the data from each object. As such, the algorithm is generated, and stored in a computer processor, which is later applied to provide the identity of an object based on the patterns in the data being received during use by a wave receiver and other transducers connected to the processor. For the purposes, the identity of an object sometimes applies to not only the object itself but also to its location in the passenger compartment. For example, a rear facing child seat is a different object than a forward facing child seat and an out-of-position adult is a different object than a normally seated adult.
To "identify" as used herein will mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
An "occupying item" of a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
"Out-of-position" as used for an occupant means that the occupant, either driver or passenger, is sufficiently close to the airbag prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. This typically occurs when the occupant's head or chest is closer than some distance such as about 5 inches from the deployment door of the airbag module. The actual distance value where airbag deployment should be suppressed depends on the design of the airbag module and is typically further for the passenger airbag than for the driver airbag.
"Dynamic out-of-position" refers to the situation where a vehicle occupant, either driver or passenger, is in position at a point in time prior to an accident but becomes out-of-position, (that is, too close to the airbag module so that he or she could be injured or killed by the deployment of the airbag,) prior to the deployment of the airbag due to pre-crash braking or other action which causes the vehicle to decelerate prior to a crash.
"Transducer" as used herein in conjunction with ultrasonics or electromagnetics will in general mean the combination of a transmitter and a receiver. In some cases, the same device will serve both as the transmitter and receiver while in others two separate devices adjacent to each other will be used. In the instant invention, a single transducer will in general be used for both sending and receiving at a particular location.
"Thermal instability" or "thermal gradients" refers to the situation where a change in air density causes a change in the path of ultrasonic waves from what the path would be in the absence of the density change. This density change ordinarily occurs due to a change in the temperature of a portion of the air through which the ultrasonic waves travel. The high speed flow of air (wind) through the passenger compartment can cause a similar effect. Thermal instability is generally caused by the sun beating down on the top of a closed vehicle ("long-term thermal instability") of through the operation of the heater or air conditioner ("short-term thermal instability"). Of course other heat sources can cause a similar effect and thus the term as used herein is not limited to the examples provided.
In the description herein on anticipatory sensing, the term "approaching" when used in connection with the mention of an object or vehicle approaching another will mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The "target" vehicle is the vehicle which is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
3. Pattern Recognition Prior Art Japanese patent 3-42337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver's mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates in locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver's eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of an automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of monitoring the position of the occupant, other that the eyes, of determining the position of the eyes relative to the passenger compartment, or of identifying any other object in the vehicle other than the driver's eyes. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant by the use of two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver's eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant, a rear facing child seat or any other object for that matter, relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with ultrasonic radiation as in the instant invention.
The use of neural networks or neural-fuzzy systems as the pattern recognition technology is central to occupant sensing since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few dozen to a hundred or so lines of code written in the C computer language as opposed to typically several hundred or more of lines when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost making them practical for automotive applications. The cost of the ultrasonic transducers, for example, is expected to be less than about $1 in automotive quantities. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than about $5 in automotive quantities of approximately 1,000,000 units per year or more.
The present invention uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction, filtering, pruning and other mathematical techniques. A non-automotive example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. "Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets", Neural Networks, Vol. 1. pp. 75-89, 1988, and "Learned Classification of Sonar Targets Using a Massively Parallel Network", IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled "Process and Apparatus for the Automatic Detection and Extraction of Features in Images and Displays" to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled "Artificial Neuron and Method of Using Same and U.S. patent application Ser. No. 08/076,601 entitled "Neural Network and Method of Using Same" to Wang, S. T. Other examples include U.S. Pat. No. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744 (Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). All of the above references are included herein by reference.
4. Ultrasonics
Ultrasonics can be used in several configurations for monitoring the interior of a passenger compartment of an automobile as described in the cross referenced patents and patent applications. In one known system, for example, two ultrasonic sensors are placed on the A-pillar and in another system, a third sensor is additionally placed in the headliner. It has been found in both of these cases that even though the proper identification is made in a high percentage of the cases, there are still a small but significant number of cases where an error in diagnosis is made based on the information received from the sensors. These systems, although a significant improvement over the other prior art, still fail to achieve the very high reliability desired by the automobile manufacturers. This shortcoming was substantially solved in Ser. No. 08/798,029 cross referenced above.
In the cases of the instant invention, and that of Ser. No. 08/798,029, as will discussed in more detail below, regardless of the number of transducers used, a trained pattern recognition system, as defined above, is used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts. This invention is particularly directed toward improving the invention of Ser. No. 08/798,029 by decreasing the sensing time, reducing the cost, improving the system response to objects which are near to the transducer mounting, and improving the ability of the system to compensate for thermal gradients and variations in the speed of sound.
5. Applications
The applications for this technology are numerous as described in the copending patent applications listed above. However, the main focus of the instant invention is for the detection of the presence of a child seat in the rear facing position or an out-of-position occupant and the detection of an occupant in a normal seating position. In the former two cases, deployment of the airbag will be suppressed and in the latter, it will be enabled.
Some examples of alternative VIMS systems follow:
In a passive infrared system one or more detectors receive infrared radiation from an object in their fields of view, in this case the vehicle occupant, and determines the temperature of the occupant based on the infrared radiation. The VIMS can then respond to the temperature of the occupant, which can either be a child in a rear facing child seat or a normally seated occupant, to control some other system. This technology could provide input data to a pattern recognition system but it has limitations related to temperature. The sensing of the child could pose a problem if the child is covered with blankets. It also might not be possible to differentiate between a rear facing child seat and a forward facing child seat. In all cases, the technology will fail to detect the occupant if the ambient temperature reaches body temperature as it does in hot climates. Nevertheless, for use in the control of the vehicle climate, for example, a passive infrared system that permits an accurate measurement of each occupant's temperature is useful.
In an optical system an infrared radiation source, frequently a light emitting diode or other laser, is used to momentarily illuminate an object, occupant or child seat in the manner as described, and illustrated in FIG. 8, of U.S. Pat. No. 5,653,462 referenced above. In some cases, a charge-coupled device (a type of TV camera also referred to as a CCD array) or a CMOS device is used to receive the reflected light. If a laser is used as the infrared light source, it can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In each case, a pattern recognition system, as defined above, is used to identify and classify, and can be used to locate, the illuminated object and its constituent parts. This system provides the most information about the object and at a rapid data rate. Its main drawback is cost which is considerably above that of ultrasonic or passive infrared systems. As the cost of infrared light sources and detectors is coming down, this system is now becoming more competitive. Depending on the implementation of the system, there may be some concern for the safety of the occupant if the laser light can enter the occupant's eyes.
Radar systems have similar properties to the infrared system discussed above. The wavelength of a particular radar system can limit the ability of the pattern recognition system to detect object features smaller than a certain size. Once again, however, there is some concern about the health effects of radar on children and other occupants. This concern is expressed in various reports available from the United States Food and Drug Administration Division of Devices.
The ultrasonic system, which is the primary focus of this invention, is the least expensive and potentially provides less information than the infrared or radar systems due to the delays resulting from the speed of sound and due to the wave length which is considerably longer than the infrared systems. The wavelength limits the detail, which can be seen by the system. Additionally, ultrasonic waves are sometimes strongly affected by thermal gradients within the vehicle such as caused by flowing air from the heater or air conditioner or as caused by the sun heating the top of the vehicle resulting in the upper part of the passenger compartment having a higher temperature than the lower part. Thermal gradients cause density changes in the air, which diffract the ultrasonic signal sending in a direction away from an object or the transducer. Although this effect has been reported in the literature no solution has been proposed prior to the present invention.
In spite of these limitations, as shown below, ultrasonics can provide sufficient timely information to permit the position and velocity of an occupant to be accurately known and, when used with an appropriate pattern recognition system, it is capable of positively determining the presence of a rear facing child seat, for example. One pattern recognition system which has been used to identify a rear facing child seat, empty seat, out-of-position occupant, etc., uses neural networks and is similar to that described in the above referenced papers by Gorman et al. Alternately, a neural-fuzzy system is now showing some promise of higher accuracy than the pure neural network system. One problem with pure neural network systems is that although the system is quite good at interpolating between vehicle passenger compartment occupancy configurations which it has been trained on, it sometimes does poorly when confronted with a totally new configuration. Neural-fuzzy systems have demonstrated the ability to better handle these situations.
A focusing system, such as used on some camera systems, could be used to determine the initial position of an occupant but is too slow to monitor his position during a crash. This is a result of the mechanical motions required to operate the lens focusing system. By itself it cannot determine the presence of a rear facing child seat or of an occupant but when used with a charge-coupled device, or CMOS array, plus some infrared illumination for night vision, and an appropriate pattern recognition system, this becomes possible.
From the above discussion, it can be seen that the addition of sophisticated pattern recognition means to any of the standard illumination and/or reception technologies for use in a motor vehicle permits the development of a host of new products, systems or capabilities heretofore not available as described in more detail below.